Posit, Python, and Quarto

Pritam Dalal and Thomas Mock

Posit and Python

Posit Team and Python

  • Workbench: develop in Python using VS Code, JupyterLab, and Jupyter Notebook (classic)

  • Connect: deploy a wide range of Python data products, including Jupyter Notebooks (reports, ETL jobs), Dash apps, FastAPIs, Quarto projects

  • Package Manager: serves up a full PyPI mirror and internally developed Python packages

Python Only Data Scientists

  • Given our deep history with R, and our extensive support for Python, Posit Team is great for bilingual data science teams.

  • It’s also an excellent soltution for Python only teams.

Posit and Python Open Source

Posit has a rich and storied history of developing open source software for the R ecosystem: RStudio IDE, tidyverse, Shiny.

In the last couple of years, Posit has begun significant development in the Python open source ecosystem with projects like Quarto, Vetiver, Shiny for Python.

Materials

Materials and Instructions

Warning!

This talk is not intended to be a self-contained workshop.

  1. install quarto

  2. install quarto VSCode extension

  3. clone github repo: https://github.com/pritamdalal/posit_python_quarto

  4. create a virtual environment and: pip install -r requirements.txt

Quarto

What is Quarto?

https://quarto.org

Quarto is an open-source scientific and technical publishing system that builds on standard markdown with features essential for scientific communication.

  • Computations: Python, R, Julia, Observable JS
  • Markdown: Pandoc flavored markdown with many enhancements
  • Output: Documents, presentations, websites, books, blogs

Literate programming system in the tradition of Org-Mode, Weave.jl, R Markdown, iPyPublish, Jupyter Book, etc.

Origins

  • Open Source project sponsored by Posit, PBC (formerly known as RStudio, PBC)

  • Builds on Posit’s decade of experience developing R Markdown, a similar system that is R-specific

  • The number of languages and runtimes used for scientific discourse is broad

  • Quarto is a ground-up re-imagining of R Markdown that is fundamentally multi-language and multi-engine

  • Quarto gets inspiration from both R Markdown and Jupyter; it allows for development in a plain-text format (.qmd) or in Jupyter notebooks (.ipynb)

Goal: Single Source Publishing

Simple Example

---
title: "matplotlib demo"
format:
  html:
    code-fold: true
jupyter: python3
---

For a demonstration of a line plot on a polar 
axis, see @fig-polar.
```{python}
#| label: fig-polar
#| fig-cap: "A line plot on a polar axis"

import numpy as np
import matplotlib.pyplot as plt

r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r
fig, ax = plt.subplots(
  subplot_kw = {'projection': 'polar'} 
)
ax.plot(theta, r)
ax.set_rticks([0.5, 1, 1.5, 2])
ax.grid(True)
plt.show()
```

Simple Example, multi-format


Can be rendered to dozens of output formats with Quarto (via Pandoc):

$ quarto render hello.qmd --to html
$ quarto render hello.qmd --to pdf
$ quarto render hello.qmd --to docx
$ quarto render hello.qmd --to epub
$ quarto render hello.qmd --to pptx
$ quarto render hello.qmd --to revealjs

Output Formats

Feature R Markdown Quarto
Basic Formats
Beamer
PowerPoint
HTML Slides
Advanced Layout
Feature R Markdown Quarto
Cross References
Websites & Blogs
Books
Interactivity Shiny Documents Quarto Interactive Documents
Paged HTML pagedown Coming soon!
Journal Articles rticles Out and more coming!
Dashboards flexdashboard Coming soon!

So what is Quarto?

Quarto® is an open-source scientific and technical publishing system built on Pandoc.

  • quarto is a language agnostic command line interface (CLI)
pritamdalal$ quarto --help
Usage:   quarto
Version: 1.2.269

Commands:
  render  [input] [args...] - Render input file(s) to various document types.            
  preview [file] [args...]  - Render and preview a document or website project.          
  publish [provider] [path] - Publish a document or project.

Basic Workflow

Rendering (execute and write to disk):

# plain text qmd
$ quarto render python.qmd
$ quarto render python.qmd --to pdf

# ipynb notebook
$ quarto render python.ipynb
$ quarto render python.ipynb --execute

Preview (execute, write to disk, and maintain a LIVE preview of content):

# plain text qmd
$ quarto preview python.qmd
$ quarto preview python.qmd --to pdf

# ipynb notebook
$ quarto preview python.ipynb
$ quarto preview python.ipynb --execute

IPython

For execution of R, Quarto uses knitr as the engine, but for Python Quarto natively executes Python with Jupyter kernels such as IPython.

  • The indicated or default Python Jupyter kernel is bound automatically when {python} executable cells are present. You can set a specific kernel via the YAML:
---
title: "My doc"
date: today
jupyter: python3
---

  • IPython executes Python code and transforms it to plain text, graphics, markdown, HTML, etc.

  • For interactive sessions, Quarto keeps the Jupyter Kernel resident as a daemon to mitigate startup times.

A .qmd is a plain text file

Metadata (YAML)

format: html
jupyter: python3
format: html
engine: knitr

Code

```{python}
from siuba import *
(mtcars
  >> group_by(_.cyl)
  >> summarize(avg_mpg = _.mpg.mean()))
```
```{r}
library(dplyr)
mtcars |> 
  group_by(cyl) |> 
  summarize(mean = mean(mpg))
```

Text

# Heading 1
This is a sentence with some **bold text**, some *italic text* and an 
![image](image.png){fig-alt="Alt text for this image"}.

Quarto can also use .ipynb as source

Rendering pipeline

Plain text workflow (.qmd uses Jupyter kernel to execute cells):

Notebook workflow (defaults to using existing stored computation):

What to do with my existing .ipynb?

You can keep using them! You get to choose whether to use the stored computation OR re-execute the document from top to bottom.


# --execute flag is optional - forces re-execution
quarto render my-favorite.ipynb --to html --execute


Quarto can help convert back and forth between plain text .qmd and .ipynb.

quarto convert --help

Usage:   quarto convert <input>
Description:
    Convert documents to alternate representations.

Convert notebook to markdown:                quarto convert doc.ipynb                
Convert markdown to notebook:                quarto convert doc.qmd                  
Convert notebook to markdown, write to file: quarto convert doc.ipynb --output doc.qmd

Develop in JupyterLab or VSCode

A screenshot of a Quarto document rendered inside JupyterLab

A screenshot of a Quarto document rendered inside VSCode

VSCode with Quarto Extension - Auto-Completion


VSCode with the Quarto extension has rich auto-completion with .qmd files:

YAML

A gif of auto-completion and search for YAML options inside RStudio

Chunk option

A gif of auto-completion of a R chunk inside RStudio

Publish Quarto to Posit Connect

quarto publish --help

  Usage:   quarto publish [provider] [path]
  Version: 1.2.269                          
                                           
  Description:
    Publish a document or project. Available providers include:
                                                               
     - Quarto Pub (quarto-pub)                                 
     - GitHub Pages (gh-pages)                                 
     - Posit Connect (connect)                               
     - Netlify (netlify)

Quarto, crafted with love and care

Development of Quarto is sponsored by Posit, PBC (formerly known as RStudio, PBC). The same core team works on both Quarto and R Markdown:

Here is the full contributors list. Quarto is open source and we welcome contributions in our github repository as well! https://github.com/quarto-dev/quarto-cli.